Method and apparatus for image processing

ABSTRACT

The present invention discloses a system and method of transforming a sample of content data by utilizing known samples in a learning best to best determine coefficients for a linear combination of non-linear filter functions and applying the coefficients to the content data in an operational phase.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application claims priority under 35 U.S.C. §119 from U.S.provisional patent application No. 60/913,493 filed Apr. 23, 2007entitled “METHOD AND APPARATUS FOR IMAGE PROCESSING” which is herebyincorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Technical Field

The invention relates to signal processing methods, particularly tocalculating values from correlated sample values (such as aneighborhood) in signal enhancement applications.

2. Background

It is a common goal in signal processing to attempt to enhance thequality of given content. Some methods exist to achieve better resultsthan simple linear convolution filters and interpolation methods such asbi-linear filters, tri-linear filters, bi-cubic filters, tri-cubicfilters, Laplace filters and Lanczos filters. Another method includes“classifying” the neighborhood using different methods (gradient,pattern matching, clustering, edge detection, optical flow) and selectspecific convolution filters, or combine convolution filters based onthat classification, such as the method described in U.S. Pat. No.6,133,957. Still another method attempts to calculate appropriate valuesbased on coarser-scale characteristics of the dataset. Some methods thentake a least square approach to determine the convolution coefficientsfor each “class” of neighborhood. In effect, all these methods aredefining non-linear filter functions of the neighborhood with someheuristic approach. Most known methods suffer from computationalcomplexity while others suffer from artifacts and mediocre imagequality.

SUMMARY OF THE DISCLOSURE

The present invention discloses a system and method of transforming asample of content data by utilizing known samples in a learning phase tobest determine coefficients for a linear combination of non-linearfilter functions and applying the coefficients to content data in anoperational phase.

In some embodiments of the present invention, the known samples aredown-sampled in the learning phase to derive target content used with atransform to determine optimal the coefficients. According to theseembodiments, the down-sampled content at least partially resembles thecontent data.

In some embodiments of the present invention, a pre-selected set offilter functions are used in a combination with a corresponding set ofweighting coefficients. According to these embodiments, the derivedcoefficients values reflect the expected symmetries and/or dependenciesof the filter functions.

In some embodiments of the present invention, the content data isobtained from capture devices such as digital still cameras or digitalvideo cameras. In some embodiments of the present invention, the contentdata is streaming video data. According to these embodiments,low-quality video content (e.g. 480i television content) is able to betransformed into high-quality television content (e.g. 1080p televisioncontent). In some other embodiments, the content data is medical imagingdata such as MRI or CT data.

In some embodiments of the present invention, an apparatus is used toaccept content data, perform the appropriate learning phase processingand operational phase processing and output transformed content data.

In some embodiments of the present invention, optional steps ofpre-emphasis, de-emphasis, memory storage, compression anddecompression, and de-aliasing are performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a process of defining optimal coefficients in alearning phase and applying the coefficients to a new sample in anoperational phase according to the present invention.

FIG. 2 illustrates an exemplary process of defining optimal coefficientsin a learning phase and applying the coefficients to a new sample in anoperational phase according to the present invention.

FIG. 3 illustrates the schematic functionality of an apparatus fordefining optimal coefficients in a learning phase and applying thecoefficients to a new sample in an operational phase according to thepresent invention.

FIG. 4 is a block diagram illustrating the exemplary image enhancementsystem in accordance with the present invention.

FIG. 5 illustrates a schematic representation a “reslet” transformtechnique in accordance with some embodiments the present invention.

DETAILED DESCRIPTION

Those of ordinary skill in the art will realize that the followingdetailed description of the present invention is illustrative only andis not intended to limit the claimed invention. Other embodiments of thepresent invention will readily suggest themselves to such skilledpersons having the benefit of this disclosure. It will be appreciatedthat in the development of any such actual implementation, numerousimplementation-specific decisions must be made in order to achieve thedeveloper's specific goals. Reference will now be made in detail toimplementations of the present invention as illustrated in theaccompanying drawings. The same reference indicators will be usedthroughout the drawings and the following detailed description to referto the same or like parts.

Given a class of discrete signals (such as a captured sample orgenerated values) that has strong correlations among a small number ofrelated samples (such as an abstract “neighborhood”; for examplespatially or temporally correlated in one or multiple dimensions in anarray), and the need for a filter operator that is a function of saidneighborhood samples plus optionally some more variables, it isdesirable to transform the discreet signals using one or more optimizedfilter functions.

In accordance with the present invention, a system and method of contenttransformation is disclosed by defining optimal coefficients in alearning phase and applying the coefficients to a new sample in anoperational phase.

Overview

FIG. 1 illustrates a process 100 of defining optimal coefficients in alearning phase and applying the coefficients to a new sample in anoperational phase according to the present invention. In the learningphase, a known sample 110 is provided representing a desired output forthe system. The known sample is then down-sampled to produce a targetsample 120. The target sample 120 is so-called because it is the targetof a transformation to determine what coefficients best reconstructs theknown sample. A transformation function (“transform”) 130 is providedthat accepts the known sample 110 values and target sample 120 values.In some embodiments of the present invention, the sample valuescorrelate to neighborhood of pixels in an array. In the preferredembodiment of the present invention, the transform 130 is a linearcombination set of pre-selected non-linear functions.

The transform 130 is solved using the known samples 110 and the targetsample 120, yielding a set of coefficients 140. The returnedcoefficients 140 are the best coefficients to reconstruct the knownsample 110 from the target sample 120.

In an operational phase, the coefficients obtained from the learningphase 150 are applied to a transform 170 along with new samples 160. Thetransform 170 is the same transform studied in the learning phase.Therefore, the output 180 is transformed to be representative of theoriginal known samples 110.

In some embodiments of the present invention, discrete samples ofdigital images, recorded digital video signals, interlaced analogtelevision signals, etc. comprise the new samples 160 that aretransformed. According these embodiments, high-quality known samples ofdigital images, digital video signals, deinterlaced digital televisionsignals, etc. are used in the training phase. Preferably the knownsamples 110 are of a very high quality such that when they aresubsequently down-sampled, the resulting target samples will emulate thenew signals. It will be readily apparent to those having ordinary skillin the art that although specific examples are given for the new samples160 and known samples 110, the present invention is equally applicableto all appropriate transformable content, now known or later developed.

In some embodiments of the present invention, a step of determining theappropriate transform 130 to perform is done manually. In otherembodiments, the appropriate transform 130 is known. In the preferredembodiment, the transform 130 is a linear combination of pre-selectednon-linear functions. The most common examples of such filter functionsare interpolations (with added parameters defining the relative positionof the continuous point within the pixel), convolutions that are linearfunctions of said neighborhood samples, or filters such as noisereduction, edge enhancement, sharpening, etc. There exists othernon-linear filter functions also commonly used, such as Xin-Li methodfor resolution enhancement, combinations of convolutions weighed bygradients. Although specific filter functions are disclosed, it will bereadily apparent to those having ordinary skill in the art that anyuseful filter function, now known or later developed, may be used forimage enhancement according to the present invention.

Exemplary Embodiment

In exemplary embodiments of the present invention, the set of filterfunctions is a set of pre-selected (not necessarily linear) primaryfunctions of the neighborhood (and optionally additional parameters).According to these embodiments, the class of functions may be defined bythe possible linear combinations of these pre-selected functions witharbitrary weights or coefficients, such as

$\begin{matrix}{y = {\sum\limits_{i = 1}^{N}{a_{i}{f_{i}( {s_{0},s_{1},s_{2},\ldots \mspace{11mu},s_{n},p_{1},\ldots \mspace{11mu},p_{m}} )}}}} & (1)\end{matrix}$

or in short,

$\begin{matrix}{y = {\sum\limits_{i = 1}^{N}{a_{i}{f_{i}( {\underset{\_}{s},\underset{\_}{p}} )}}}} & (2)\end{matrix}$

where s_(i) (1≦i≦n) are the discrete values of a given neighborhood ofs₀ and p=p₁,p₂, . . . ,p_(m) are optional additional parameters, andƒ_(i)(s,p) (1≦i≦N) are the pre-selected primary filter functions. Eachspecific set of a_(i) coefficients define a specific filter functionwithin this class.

In some embodiments, the optional parameters, p=p₁,p₂, . . . ,p_(m)create an f(x,y) or f(x,y,z) function that would best approximate thecontinuous original source within the sample where z may is a thirddimension value in space or time. In some embodiments, the optionalparameters allow a single filter to be trained to perform arbitraryscaling/upscaling/zooming even to non-integer factors, or changing theframe rate (if the z denotes the time scale in case of a video signal)arbitrarily by a single filter.

The training set is a large set of samples along with their respectiveknown desired y values that we wish to approximate with a filterfunction. The system can then take the training set in the trainingphase, and find the coefficients such that the difference of the desiredy values and the value of the F(s, p) is the smallest in some metrics;and then use the filter defined by those coefficients in the operationalphase. There are many options including iterative, Gauss, neural networkand other methods to accomplish this.

One preferred method described below is regression, that can find thebest coefficients for the above defined class of filter functions sothat the specific function within that class defined by thosecoefficients will best approximate the a-priori desired values of thesaid sample in “mean square error” or PSNR (peak signal to noise ratio)terms.

According to this exemplary embodiment, for a given training set,consisting of a large set of s_(o) ^(j) and corresponding s_(i) ^(j)(1≦j≦M) (1≦i≦n) and p^(j) parameter values with desired y^(j) values wewant to determine the best possible a_(i) coefficients.

With the notation s′_(i) ^(j)=ƒ_(i)(s₀ ^(j),s₁ ^(j),s₂ ^(j), . . .,s_(n) ^(j),p^(j)) (1≦i≦N) one possible method is to create the matrix

$S = {{\begin{bmatrix}s_{1}^{\prime 1} & s_{2}^{\prime 1} & \cdots & s_{N}^{\prime 1} \\s_{1}^{\prime 2} & s_{2}^{\prime 2} & \cdots & s_{N}^{\prime 2} \\\cdots & \cdots & \cdots & \cdots \\s_{1}^{\prime \; M} & s_{2}^{\prime \; M} & \cdots & s_{N}^{\prime \; M}\end{bmatrix}\mspace{14mu} {and}\mspace{14mu} {the}\mspace{14mu} {vector}\mspace{14mu} \underset{\_}{y}} = \begin{bmatrix}y^{1} \\y^{2} \\\cdots \\y^{M}\end{bmatrix}}$

with the unknown coefficients being

$\underset{\_}{a} = \begin{bmatrix}a_{1} \\a_{2} \\\vdots \\a_{N}\end{bmatrix}$

and then the problem can be formulated as solving the system of linearequations

${\begin{bmatrix}s_{1}^{\prime 1} & s_{2}^{\prime 1} & \cdots & s_{N}^{\prime 1} \\s_{1}^{\prime 2} & s_{2}^{\prime 2} & \cdots & s_{N}^{\prime 2} \\\cdots & \cdots & \cdots & \cdots \\s_{1}^{\prime \; M} & s_{2}^{\prime \; M} & \cdots & s_{N}^{\prime \; M}\end{bmatrix}\begin{bmatrix}a_{1} \\a_{2} \\\vdots \\a_{N}\end{bmatrix}} = \begin{bmatrix}y^{1} \\y^{2} \\\cdots \\y^{M}\end{bmatrix}$

-   -   or in short,

Sa=y   (3)

With M>N this system of linear equations is overdetermined, but we canfind the solution where the length of Sa−y is minimal, which isequivalent of minimizing the mean square error. Also, Sa−y is minimal ifand only if it is perpendicular to the N dimensional subspace of the Mdimensional Euclidean space defined by S.

One possible solution based on these considerations is given by theMoore-Penrose pseudo-inverse, the best x in those terms can be obtainedby

S ^(T)(Sa−y)=0

S^(T)Sa=S^(T) y

a=(S ^(T) S)⁻¹ S ^(T) y   (4)

While creating the large N×M matrix is computationally too complex withthe desired large training set of samples, we can dynamically maintainthe N×N matrix S^(T) S and the N dimensional vector S^(T) y in thelearning phase by starting with a nullmatrix and nullvector respectivelyand for each sample in the training set adding the diadmatrix s′^(jT)s′^(j) to S^(T) S and add s′^(jT) y^(j) to S^(T) y where (1≦j≦M). Afterthe learning phase, we can calculate a vector (that is the coefficients)from the Moore-Penrose formula, and use them in the operational phase.If the S^(T) S matrix has no inverse, that is the system of linearequations is underdetermined, we can still find one of the infinitesolutions of the equation S^(T) Sa=S^(T) y.

When multiple filters are required for some image processing application(such as color demosaicing (one for each color plane), or resolutionenhancement (one per subpixel), or when processing color images) twopossible options are available. In one embodiment of this invention,each of the I filters can be treated individually and independently. Inanother embodiment of the invention, a single filter can be used thattakes I*(n+1) samples with the following notation:

$\begin{matrix}{{F_{1}( {s_{0},s_{1},\ldots \;,s_{n}} )} = {F( {s_{0},s_{1},\ldots \;,s_{n},\overset{n + 1}{0},\overset{n + 2}{0},\ldots \;,\overset{{l*{({n + 1})}} - 1}{0}} )}} \\{{F_{2}( {s_{0},s_{1},\ldots \;,s_{n}} )} = {F( {\overset{0}{0},\overset{1}{0},\ldots \;,\overset{n}{0},s_{0},s_{1},\ldots \;,s_{n},\overset{{2n} + 2}{0},\ldots \;,\overset{{l*{({n + 1})}} - 1}{0}} )}} \\\cdots \\{{F_{l}( {s_{0},s_{1},\ldots \;,s_{n}} )} = {F( {\overset{0}{0},\overset{1}{0},\ldots \;,\overset{{{({l - 1})}{({n + 1})}} - 1}{0},s_{0},s_{1},\ldots \;,s_{n}} )}}\end{matrix}$

The advantage of this method is that expected symmetries orcodependencies of the different F_(i) functions can be coded into thecoefficients by selecting the f functions reflecting the expectedsymmetries or dependencies. Or, in general, if the a coefficients have anumber of symmetry and other properties (for example that the sum of thesubpixels should be equal to s₀), that can be formulated in L linearequations such as

Ca=b

where L<N, the formula can be solved as

$C^{\prime} = {{\begin{bmatrix}1 & 0 & \cdots & 0 \\0 & 1 & \; & 0 \\\cdots & \; & \; & 0 \\0 & \cdots & \; & 0 \\c_{1,1} & c_{1,2} & \cdots & c_{1,N} \\\cdots & \; & \; & \; \\c_{l,1} & c_{l,2} & \cdots & c_{l,N}\end{bmatrix}\mspace{14mu} ( {{size}\text{:}\mspace{11mu} N \times N} )\mspace{14mu} {\underset{\_}{b}}^{\prime}} = {\begin{bmatrix}0 \\0 \\\cdots \\0 \\b_{1} \\\cdots \\b_{L}\end{bmatrix}\mspace{14mu} ( {{size}\text{:}\mspace{11mu} N} )}}$$E = {{\begin{bmatrix}1 & 0 & \cdots & 0 \\0 & 1 & \; & 0 \\\cdots & \; & \; & 0 \\0 & \cdots & \; & 1 \\0 & 0 & \cdots & 0 \\\cdots & \; & \; & \; \\0 & 0 & \cdots & 0\end{bmatrix}\mspace{14mu} ( {{{size}\text{:}\mspace{11mu} N} - {L \times N}} )\mspace{14mu} {and}\mspace{14mu} {\underset{\_}{a}}^{\prime}} = {\begin{bmatrix}a_{1} \\a_{2} \\\cdots \\\; \\a_{N - L}\end{bmatrix}\mspace{14mu} ( {N - L} )}}$C′a=Ea′+b′

a=C′ ⁻¹ Ea′+C′⁻¹ b′

from which the original equation (3) is rewritten as

Sa=y

S(C′ ⁻¹ Ea′+C′ ⁻¹ b′)=y

SC′ ⁻¹ Ea′=y−C′ ⁻¹ b′

with, S′=SC′⁻¹ E and y′=y−C′⁻¹ b′ substitutions yield the S′a′=y′overdetermined set of equations with S′ being L×M in size which can bethen solved as (3).

Most existing solutions using non-linear filters can be regarded as aspecific case of the above class of F filter functions. For example,those that classify the signal based on the neighborhood (usually usinga gradient filter) can be regarded as selecting a set ofƒ_(i,j)(s)=ƒ′_(j)(s)s_(i) functions yielding

$\begin{matrix}{{y = {\sum{{f_{j}^{\prime}( \underset{\_}{s} )}( {{a_{0,j}s_{0}} + {a_{1,j}s_{1}} + \ldots + {a_{n,j}s_{n}}} )}}}{y = {\sum\limits_{j}( {{f_{j}^{\prime}( \underset{\_}{s} )}{\sum\limits_{i}{a_{i,j}s_{i}}}} )}}} & (5)\end{matrix}$

such that for each s exactly one of the ƒ′_(j)(s) functions are 1, andthe other ones are 0. The problem with those methods is that theresulting F function is not continuous, causing artifacts when a largearea of samples of one class borders with another area of another class.To address continuity, in some variations, ƒ′_(j)(s) are not 1 or 0, butdenote the “confidence” factor of s₀ and its neighborhood belonging to aspecific class. The advantage of these methods is that only a relativelysmall number of non-linear f functions need to be computed for eachsample neighborhood.

One consideration is that in the majority of the applications the filterfunction is expected to have the following invariance property(homogeneity):

F(αs ₀ +β,αs ₁ +β, . . . ,αs _(n) +β,p ₁ , . . . ,p _(m))=αF(s ₀ ,s ₁ ,. . . ,s _(n) ,p ₁ , . . . ,p _(m))+β  (6)

It is easy to see that this can be accomplished with (5) only if it canbe rewritten as follows:

$\begin{matrix}{y = {s_{0} + {\sum\limits_{j}( {{f_{j}^{''}( \underset{\_}{s} )}{\sum\limits_{i}{a_{i,j}( {s_{i} - s_{0}} )}}} )}}} & (7)\end{matrix}$

where each ƒ″ function has the following property:

ƒ″(αs ₀ +β,αs ₁ +β, . . . ,αs _(n)+β)=ƒ″(s ₀ ,s ₁ , . . . ,s _(n))   (8)

One approach therefore is to select f functions with such properties.Heuristic approaches such as gradient based methods or classificationscomply with this constraint.

The concern remains if (7) may limit how well the ideal F function withthe constraint of (6) can be approximated.

To address that concern, let's suppose we have a d “norm” function withthe following property:

-   d(αs)=|α|d(s)—examples of such functions include the norm functions    d(s)=√{square root over (Σs_(i) ²)} or in general, d(s)=^(k)√{square    root over (c_(i)Σs_(i) ^(k))} or d(s)=max(c_(i)|s_(i)|)    It is easy to prove that (6) is true for a continuous F only if F    can be rewritten as follows:

${F( {s_{0},s_{1},\ldots \mspace{11mu},s_{n},p_{1},\ldots \mspace{11mu},p_{m}} )} = \{ \begin{matrix}{ s_{0} \middle| {\forall{i\mspace{11mu} \text{:}\mspace{11mu} s_{i}}}  = s_{0}} \\ {s_{0} + {{d^{\prime}( \underset{\_}{s} )}{F^{\prime}\begin{pmatrix}{\frac{s_{1} - s_{0}}{d^{\prime}( \underset{\_}{s} )},\frac{s_{2} - s_{0}}{d^{\prime}( \underset{\_}{s} )},\ldots \;,} \\{\frac{s_{n} - s_{0}}{d^{\prime}( \underset{\_}{s} )},p_{1},\ldots \;,p_{m}}\end{pmatrix}}}} \middle| {otherwise} \end{matrix} $

where d′(s)=d(s₁−s₀,s₂−s₀, . . . ,s_(n)−s₀) and F′ is an odd functionfor the s variables such that F′(s₀,s₁, . . . ,s_(n),p₁, . . .,p_(m))=−F′(−s₀,−s₁, . . . ,−s_(n),p₁, . . . ,p_(m)).

As the most generic approach, F′ can be well approximated withpolynomials. As F′ should be odd, it should only include the monomialsof odd orders, so with the specific choice of d′(s)=√{square root over(Σ(s_(i)−s₀)²)} F it can be expressed as:

${F( {s_{0},s_{1},\ldots \;,s_{n}} )} = {{s_{0} + {\sum{a_{i}( {s_{i} - s_{0}} )}} + {\sum{a_{i,j,k}\frac{( {s_{i} - s_{0}} )( {s_{j} - s_{0}} )( {s_{k} - s_{0}} )}{\sum( {s_{i} - s_{0}} )^{2}}}} + \ldots} = {s_{0} + {\sum{a_{i}( {s_{i} - s_{0}} )}} + {\sum\limits_{j,k}{\frac{( {s_{j} - s_{0}} )( {s_{k} - s_{0}} )}{\sum( {s_{i} - s_{0}} )^{2}}( {\sum\limits_{i}{a_{i,j,k}( {s_{i} - s_{0}} )}} )}} + \ldots}}$

which is exactly (7) with the special choice of f functions:

${{f_{0}^{\prime}( \underset{\_}{s} )} = 1},{{f_{j,k}^{\prime}( \underset{\_}{s} )} = \frac{( {s_{j} - s_{0}} )( {s_{k} - s_{0}} )}{\sum( {s_{i} - s_{0}} )}},{{f_{j,k,l,m}^{\prime}( \underset{\_}{s} )} = \frac{( {s_{j} - s_{0}} )( {s_{k} - s_{0}} )( {s_{l} - s_{0}} )( {s_{m} - s_{0}} )}{( {\sum( {s_{i} - s_{0}} )^{2}} )^{2}}},\ldots$

-   -   This proves that filters in the form of (7) can approximate the        ideal F function arbitrarily well.

However, this formula is still too generic, yielding too manycoefficients and respective monomials even at order 3. One pruningoption is to consider approximation only until order 1 monomials whichyields the class of well known convolutions. A natural extension oflinear convolutions therefore is selecting all, or selected monomialsfor j, k gives us f′ functions

${f_{j,k}( \underset{\_}{s} )} = \frac{( {s_{j} - s_{0}} )( {s_{k} - s_{0}} )}{\sum( {s_{i} - s_{0}} )}$

Note that the fact that the norm function needs to be calculated onlyonce per sample and neighborhood and is reused in each f primaryfunction further reduces computational complexity.

Other good choices for the f′ functions at order 3 approximations afterthese considerations (using other reformulations or different normfunctions) are

$\begin{matrix}{{\frac{\sum{c_{ij}{{s_{i} - s_{j}}}^{k}}}{\sum{c_{i}{{s_{i} - s_{0}}}^{k}}}\mspace{14mu} {or}\mspace{14mu} \frac{\sum{c_{ij}{{s_{i} - s_{j}}}^{k}}}{\max ( {c_{i}{{s_{i} - s_{0}}}^{k}} )}\mspace{14mu} {for}\mspace{14mu} {some}\mspace{14mu} k} \in N} & (9)\end{matrix}$

Let's notice that specific choices of i, j pairs will generate functionsthat will in a sense measure “directions” in arrays; such as where thecorresponding samples line up in some direction in the neighborhood ofs₀. However, instead of heuristics used in earlier methods, thisapparatus can take a statistical approach and simply identify the bestsuch monomials and the best linear combinations thereof from puremathematical and statistical considerations without making thosea-priori assumptions about directions and gradients; therefore it is notlimited to image processing.

The norm function may vary depending on the type of the correlation ofthe neighborhood. When the signal is a color array or a color mosaic,there are different approaches:

-   -   a) treat each color channel as separate problems    -   b) treat each color channel as the third (fourth) dimension in        the neighborhood    -   c) treat the problem as each s is a vector rather than a scalar        value

Approach a) is a good approach although it may not be able to exploitcertain correlations among the color channels. b) and c) are formallydifferent, but technically very similar; the choice of a good “norm”function is key. Consider s_(i)=(s_(i) ^(r),s_(i) ^(g),s_(i) ^(b)) inthe case of a color array. One good approach may be to consider normsadapted to the correlations such as

${d^{\prime \; r}( \underset{\_}{s} )} = \sqrt[k]{{c_{i}^{r,r}{\sum( {s_{i}^{r} - s_{0}^{r}} )^{k}}} + {c_{i}^{r,g}{\sum( {s_{i}^{g} - s_{0}^{g}} )^{k}}} + {c_{i}^{r,b}{\sum( {s_{i}^{b} - s_{0}^{b}} )^{k}}}}$

and in general only subtract sample values belonging to the same channelin (9). Similar considerations should be used with other type ofcorrelations, such as a color mosaic obtained by a Bayer filter.

Illustrative Example

In an illustrative example, a known high resolution progressive scantelevision sample is utilized in the learning phase and determines thebest coefficients to be used in an operational phase for deinterlacing alow resolution interlaced television signal. According to FIG. 2, aknown high resolution progressive scan television signal (1080p) sample210 is provided in a learning phase. The 1080p sample 210 isdown-sampled to a low resolution interlaced (480i) target sample 220.Knowing that a transform comprising a combination of deinterlacing andresolution enhancing functions must be performed to reconstruct the1080p sample 210 from the 480i sample, the known values of the 1080isample 210 and the 480i sample are applied to the transform 230 todetermine optimal coefficients 240 for the transform 230.

Once the learning phase finds the optimal coefficients 240 they are usedin the operational phase. A new low resolution interlaced sample (New480i) 260 and the optimal coefficients 250 are applied to the transform270. The transform 270 is the same set of deinterlacing and resolutionenhancing functions as transform 230. Therefore, the resultant output ofthe transform 230 is a new high resolution progressive scan televisionsignal (New 1080p) 280 representative of the original 1080p sample.

System for Processing

The process of using a training set of data to optimize the filterfunction to be applied to a target signal may be incorporated within acomputing system according to some embodiments of the present invention.The schematic functionality of this apparatus is represented in FIG. 3.

One exemplary apparatus whose functionality is described in FIG. 3 isportrayed in FIG. 4. FIG. 4 is a block diagram illustrating theexemplary image enhancement system 400 in accordance with the presentinvention. The system 400 is comprised of a content transform apparatus425 used to transform the content from sources 401, 402, 403, 404 and/or405 and output a transformed sample. In some embodiments of the presentinvention, more than one content-type is able to be transformed by thecontent transform apparatus 425. In some embodiments, content source 401is a still image capture device, such as a digital camera having a Bayerfilter or another appropriate color filter, content source 402 is adigital video capture device, content source 403 is a streaming videosource, content source 404 is a computer-generated video source andcontent source 405 is 3-dimensional medical imaging device such as a MRIdevice. It will be readily apparent to those having ordinary skill inthe art that, although specific examples of signal sources are given,any appropriate signal source, now known or later developed, maysimilarly be processed by the content transform apparatus 425 of thepresent invention.

In some embodiments of the present invention, content from the contentsources 401, 402, 403, 404 and/or 405 is gathered by the raw datacollector 410. Also, in some embodiments, the data is stored in apre-processing memory device 415. Furthermore, a pre-processor 420 isutilized in some embodiments of the present invention to conductpreliminary processing on data from the content sources 401, 402, 403,404 and/or 405. In some embodiments of the present invention, thepreliminary processing includes performing an anti-aliasing step on thedata. In some embodiments, pre-processing compression or decompressionis performed on the data in the pre-processor. In yet other embodimentsof the present invention, multiple pre-processing steps are conducted onthe signal data in the pre-processor 420. Furthermore, although specificexamples of pre-processing are disclosed, it will be readily apparent tothose having ordinary skill in the art that any appropriatepre-processing function, now known or later developed, may beaccomplished by the pre-processor 420.

In some embodiments of the present invention, the content transformapparatus 425 includes: a known sample memory 430, a down-samplingprocessor 435, a target sample memory 440 and a learning phase processor445. According to the method described above, known samples from theknown sample memory 430 are down-sampled with the down-samplingprocessor 435, resulting in target samples, which are stored in thetarget sample memory 440. Known samples and target samples are accessedfrom the known sample memory 430 and the target sample memory,respectively by the learning phase processor 445. Likewise, transformsare loaded into the learning phase processor 445 from the transformmemory 450 and a learning phase step is performed, resulting in knownoptimal coefficient values. The known optimal coefficient values arestored in a coefficient memory 455.

Preferably, the actions performed by the content transform apparatus 425are controlled by a controller 424. In some embodiments of the presentinvention, the controller 424 is coupled to a user interface 423.

Upon determining optimal coefficient values in the learning phase, theoperational phase processor 460 loads the same transforms form thelearning phase from the transform memory 450, as well as thecoefficients from the coefficient memory 455. Furthermore, content datais delivered to the operational phase processor 460 from one or morecontent source 401, from the raw data collector 410, from thepre-processing memory device 415 and/or the pre-processor 420. Uponloading, an operational phase transform is performed, resulting intransformed output data representative of the known samples.

In some embodiments of the present invention, at least one non-linearfilter function is used during the learning phase and the operationalphase. In the preferred embodiment of the present invention, learningphase and the operational phase utilize the transform described in theexemplary embodiment from above, as defined by equations (1)-(9).

In some embodiments of the present invention, the transformed outputdata is directly outputted to output 470. In other embodiments, apost-processor 465 is provided to perform ancillary processing on theenhanced signal data. Once outputted, the transformed output data isstored in an output memory 480 or sent directly to output devices 496,497, 498 and/or 499, or both. According to some embodiments, outputdevice 496 is a personal data assistant (PDA) or Smartphone (such as aBlackberry™ device manufactured by RIM® or the iphone™ manufactured byApple®), output device 497 is a computer or television monitor, outputdevice 498 is a medical apparatus imaging device, and output 499 is astreaming video signal available over a computer network, such as theinternet. Although specific examples of output devices are disclosed, itwill be readily apparent to those having ordinary skill in the art thatany appropriate output device, now known or later developed, isenvisioned for use as an output device according to the presentinvention. In some embodiments, the enhanced signal is outputted to anadditional peripheral device 485.

“Reslet” Transformations

In some embodiments of the present invention, a “reslet” transformationis utilized for content enhancement. The effectiveness of wavelet basedcompression methods currently used take advantage of the fact that asubsampled image (using a low-pass filter) can be reconstructed to animage of the original resolution with an appropriate inverse filter; andthe difference from this approximation is equivalent of a high-passfilter; from the combination of the two datasets the original image canbe properly reconstructed. The difference tends to be minimal for themajority of the samples, which decorrelates the image well, making thistransformation a good candidate for image compression. This view howeverforecasts that better methods that approximate the original image fromthe subsampled version will produce better compression results.According to some embodiments of the present invention, a resolutionenhancement technique is disclosed that can produce a good approximationof the original image from a subsampled set (by a 2× factor in bothdimensions).

FIG. 5 illustrates a schematic representation a “reslet” transformtechnique. Given a resolution enhancement technique that can produce agood approximation of the original image from a subsampled set (by a 2×factor in both dimensions) one step of the transform is the following:

The top left quarter of the image becomes the subsampled image using the“box convolution kernel”, that is each pixel is obtained by the averageor the sum of the 2×2 original pixels of the high-resolution image.

The top right corner is obtained by the “delta”, the difference betweenthe original top right pixel of each corresponding low-res pixel, andthe approximation by the resolution enhancement filter. Similarly, thebottom left and bottom right pixels are the error of the approximationmethod for the corresponding subpixels.

Then, similarly to a Wavelet transform, the top left subsampled image isrecursively transformed again and again for a preset number ofiterations or until a preset threshold size is reached.

For the inverse transformation in each step the large image isapproximated from the top left subsampled image; then the delta isapplied for the top right, bottom left and bottom right subpixels. Thereis no “delta” for the top left subpixel, but at the last step, as thesum of the 4 pixels should be equal to 4× the subsampled pixel, the topleft pixel value should be adjusted accordingly. This will fully restorethe original image without rounding errors.

The wavelet transformation of any wavelet-based compression methods canbe replaced by the above defined “reslet” transformation, and thereforeany known compression method can be used.

The present invention has been described in terms of specificembodiments incorporating details to facilitate the understanding of theprinciples of construction and operation of the invention. Suchreference herein to specific embodiments and details thereof is notintended to limit the scope of the claims appended hereto. It will beapparent to those skilled in the art that modifications can be made inthe embodiment chosen for illustration without departing from the spiritand scope of the invention. Specifically, it will be apparent to one ofordinary skill in the art that the device and method of the presentinvention could be implemented in several different ways and haveseveral different appearances.

1. An apparatus for content processing filtering in an operational phasethat is capable of learning from a training set of known discretecontent signals and desired transform values by finding the bestcoefficients for a transform in a learning phase, the transformcomprising linear combinations of pre-selected non-linear functions. 2.The apparatus for content processing filtering according to claim 1,wherein the transform allows fast calculations in the operational phase.3. The apparatus for content processing filtering according to claim 1,wherein the transform is in the format of:y=a ₀ s ₀ +a ₁ s ₁ + . . . +a _(n) s _(n)+ƒ₁(s)(a _(1,0) s ₀ +a _(1,1) s₁ + . . . +a _(1,n) s _(n))+ƒ₂(s)(a _(2,0) s ₀ +a _(2,1) s ₁ + . . . +a_(2,n) s _(n))+ . . . .
 4. The apparatus for content processingfiltering according to claim 3, wherein the functions have the propertyof ƒ(as+c)=ƒ(s) for each scalar a and c values where s+c denotes thevector (s₀+c,s₁+c, . . .,s_(n)+c), and${\sum\limits_{0 \leq i \leq n}a_{i}} = 1$ as well as${\sum\limits_{0 \leq i \leq n}a_{k,i}} = 1$ , and the desired filterfunction is in the format of: F(as+c)=aF(s)+c, wherein the linearcombination is rearranged as follows, biased to s₀: y−s₀=a₁(s₁−s₀)+ . .. +a_(n)(s_(n)−s₀)+ƒ₁(s)(a_(1,1)(s₁−s₀+ . . . +a_(1,n)(s_(n)−s₀))+ . . ., thus optimizing this biased variation.
 5. The apparatus for contentprocessing filtering according to claim 3, wherein the functions havethe format of:${\frac{\sum{c_{ij}{{s_{i} - s_{j}}}^{k}}}{\sum{c_{i}{{s_{i} - s_{0}}}^{k}}}\mspace{14mu} {or}\mspace{14mu} \frac{\sum{c_{ij}{{s_{i} - s_{j}}}^{k}}}{\max ( {c_{i}{{s_{i} - s_{0}}}^{k}} )}\mspace{14mu} {for}\mspace{14mu} {some}\mspace{14mu} k} \in {N.}$6. The apparatus for content processing filtering according to claim 5,wherein the content comprises a neighborhood of pixels, and wherein theneighborhood of pixels is specifically defined by a set of pixels in theproximity of a base pixel s₀ wherein the functions have nonzero c_(ij)values only for i,j pairs that line up in a specific direction.
 7. Theapparatus for content processing filtering according to claim 6, whereinthe neighborhood of pixels is selected from among a 3×3 neighborhood ofa two dimensional array of pixels, a 5×5 neighborhood of a twodimensional array of pixels, a 3×3×3 neighborhood of a three dimensionalarray of pixels and a 5×5×5 neighborhood of a three dimensional array ofpixels.
 8. The apparatus for content processing filtering according toclaim 6, wherein the neighborhood of pixels is selected from among a m×mneighborhood of a two dimensional array of pixels and a m×m×mneighborhood of a three dimensional array of pixels, wherein m is aninteger.
 9. The apparatus for content processing filtering according toclaim 6, wherein the discrete content signal is selected from among asound signal, a image signal and a video signal, and wherein the filterfurther comprises: a. a continuous sample value with a point defined bythe coordinates x,y[, z . . . ] within a set of pixels; and b. subpixelswithin the set of pixels, in the form of n*k[*m . . . ] supersampling;wherein the training set is created by downscaling the discrete contentsignal, resulting in downscaled content, and providing the downscaledcontent and each subpixel or specific values at x,y[,z . . . ] of theoriginal discrete content signals for the learning process, using linearcombinations of arbitrarily selected non-linear filters.
 10. Theapparatus for content processing filtering according to claim 1, whereinthe discrete content signal is three dimensional in space, and whereinthe content processing filtering comprises adding slices to anon-isotropic medical dataset for optimal three dimensional volumereconstruction.
 11. The apparatus for content processing filteringaccording to claim 10, wherein the medical dataset is selected fromamong MRI image data and CT image data.
 12. The apparatus for contentprocessing filtering according to claim 9, wherein the content is usedin consumer electronic devices.
 13. The apparatus for content processingfiltering according to claim 12, wherein the consumer electronic devicesare selected from among digital cameras and scanners.
 14. The apparatusfor content processing filtering according to claim 1, wherein thecontent is a video stream, wherein the video stream is three-dimensionalin time and space, and wherein all three dimensions are utilized fortransforming two dimensions of the content.
 15. The apparatus forcontent processing filtering according to claim 4, wherein the contentis obtained from a capture device utilizing a Bayer filter, and whereinthe content is transformed with filter values for demosaicing thecontent.
 16. The apparatus for content processing filtering according toclaim 15, wherein the content comprises a neighborhood of pixels, andwherein the neighborhood of pixels is specifically defined by a set ofpixels in the format of: g_(−2,−2) b_(−2,−1) g_(−2,0) b_(−2,1) g_(−2,2)r_(−1,−2) g_(−1,−1) r_(−1,0) g_(−1,1) r_(−1,2) g_(0,−2) b_(0,−1) g_(0,0)b_(0,1) g_(0,2) r_(1,−2) g_(1,−1) r_(1,0) g_(1,1) r_(1,2) g_(2,−2)b_(2,−1) g_(2,0) b_(2,1) g_(2,2) wherein$r_{0,0}^{\prime} = \frac{r_{{- 1},0} - r_{1,0}}{2}$$b_{0,0}^{\prime} = \frac{b_{0,{- 1}} - b_{0,1}}{2}$r_(0, 0) = r_(0, 0)^(′) + ∑a_(i, j)^(′)(g_(i, j) − g_(0, 0)) + ∑a_(i, j)^(r)(r_(i, j) − r_(0, 0)^(′)) + ∑a_(i, j)^(b)(b_(i, j) − b_(0, 0)^(′)) + f(s)(∑a_(i, j)^(g)(g_(i, j) − g_(0, 0)) + ∑a_(i, j)^(r)(r_(i, j) − r_(0, 0)) + ∑a_(i, j)^(b)(b_(i, j) − b_(0, 0))) + …and wherein the transform is in the format of: $\frac{\begin{matrix}{{\alpha ( {g_{{- 1},{- 1}} - {g_{0,0}}^{2} + ( {g_{0,0} - g_{1,1}} )^{2}} )} +} \\{{\beta ( {g_{{- 1},{- 1}} - g_{1,1}} )}^{2} + {\gamma( {( {r_{{- 1},0} - r_{1,{- 2}}} )^{2} +} }} \\ {( {r_{{- 1},2} - r_{1,0}} )^{2} + ( {b_{{- 2},1} - b_{0,{- 1}}} )^{2} + ( {b_{0,1} - b_{2,{- 1}}} )^{2}} )\end{matrix}}{\begin{matrix}{{\sum{a_{i,j}^{g}( {g_{i,j} - g_{0,0}} )}^{2}} +} \\{{\sum{a_{i,j}^{r}( {r_{i,j} - r_{0,0}^{\prime}} )^{2}}} + {\sum{a_{i,j}^{r}( {b_{i,j} - b_{0,0}^{\prime}} )}^{2}}}\end{matrix}}$
 17. The apparatus for content processing filteringaccording to claim 1, wherein the transform removes artifacts of contentsignals that turn noisy, such as transmitted over noisy channels ordigital compression, wherein the transform is taught using a trainingset containing corresponding noisy and original images in the learningphase.
 18. The apparatus for content processing filtering according toclaim 1, wherein the transform comprises image enhancement filtersselected from among filters for edge enhancement, sharpening, edgedetection and zooming.
 19. The apparatus for content processingfiltering according to claim 1, further comprising: a pre-emphasis phaseand a de-emphasis phase for both the learning phase and the operationalphase, wherein the pre-emphasis phase and the de-emphasis phase utilizean appropriate filter pair.
 20. The apparatus for content processingfiltering according to claim 19, wherein the filter pair comprises anedge enhancement linear convolution filter and its inverse convolution.21. A method of transforming content data using a learning phase and aoperational phase comprising: a training phase comprising: providing atleast one sample of high-quality content and a transform, wherein thetransform is defined by a linear combination of pre-selected non-linearfunctions weighted with coefficients; the at least one sample ofhigh-quality content to produce a degraded quality content, resulting ina representative target sample; applying the transform to therepresentative target sample; and determining values for thecoefficients that will best transform the representative target sampleto resemble the at least one sample of high-quality content, resultingin optimized coefficients; and an operational phase comprising:providing at least one new sample; and applying the transform and theoptimized coefficients to the at least one new sample, resulting inenhanced content data.
 22. The method of transforming content dataaccording to claim 21, wherein the content data is image data capturedwith a camera using a Bayer filter.
 23. The method of transformingcontent data according to claim 22, wherein the transform at leastpartially comprises a filter for demosaicing.
 24. The method oftransforming content data according to claim 21, wherein the contentdata is image data, and wherein the transform at least partiallycomprises a filter for performing a function selected from among:resolution enhancement, filtering, noise reduction, color filterdemosaicing, zooming, edge enhancement and sharpening.
 25. The method oftransforming content data according to claim 21, wherein the contentdata is medical imaging data.
 26. The method of transforming contentdata according to claim 21, the method further comprising a pre-emphasisphase and a de-emphasis phase.
 27. A system for transforming contentdata comprising: at least one content data source; an image processingapparatus comprising: a transform memory containing a set of at leastone pre-selected filter function weighted with at least one coefficient;a known data memory containing at least one known content sample; atarget sample memory; a down-sampling processor, wherein thedown-sampling processor obtains a known content sample from the knowndata memory and performs a down-sampling transform, resulting in atarget sample; a target sample memory; a learning phase processor,wherein the learning phase processor obtains a known content sample, atleast one pre-selected filter function, and a target sample, and whereinthe learning phase processor determines the best coefficients fortransforming the target sample to the known content sample, resulting inat least one optimal coefficient; and an operational phase processor,wherein the operational phase processor obtains at least one preselectedfilter function, at least one optimal coefficient and content data fromthe at least one content data source, and wherein the content data istransformed using the at least one filter function and the at least oneoptimal coefficient, resulting in transformed content data; and anoutput.
 28. The system for transforming content data according to claim27, wherein the at least one pre-selected filter function comprises alinear combination of pre-selected non-linear functions.
 29. The systemfor transforming content data according to claim 27, wherein at leastone content data source is selected from among: image capture devices,scanners, video capture devices, medical imaging devices, streamingvideo sources, and generated video sources.
 30. The system fortransforming content data according to claim 27, wherein the imageprocessing apparatus is incorporated into a television cable modem,wherein the at least one content source is a low resolution televisionsignal, wherein the set of at least one pre-selected filter functioncomprises at least partially a resolution enhancement function, andwherein the output is a high resolution television signal.
 31. Thesystem for transforming content data according to claim 30, wherein thelow resolution television signal comprises a low resolution interlacedtelevision signal, wherein the set of at least one pre-selected filterfunction comprises at least partially a deinterlacing function, andwherein the output is a high resolution progressive scan televisionsignal.
 32. The apparatus for content processing filtering according toclaim 1, further comprising a system for compressing content data usingreslet techniques where a low-pass filter and a high-pass filter of awavelet method is replaced with a down-sampling and deltas to the resletmethod.
 33. The apparatus for content processing filtering according toclaim 1, further comprising a system for compressing content data usingreslet techniques where a low-pass filter and a high-pass filter of awavelet method is replaced with a down-sampling and deltas to the resletmethod.